Plant recognition includes two kinds of tasks:specimen recognition and real-environment recognition. Due to the existence of background noise, real-environment plant image recognition is more difficult. To reduce the weight of Convolutional Neural Networks (CNN), to improve over-fitting, to improve the recognition rate and generalization ability, a method of plant identification with Family Priority (FP) was proposed. Combined with the lightweight CNN MobileNet model, a plant recognition model Family Priority MobileNet (FP-MobileNet) was established by means of migration learning. On the single background plant dataset flavia, the MobileNet model achieved 99.8% of accuracy. For the more challenging real-environment flower dataset flower102, when the number of samples in the training set was greater than that in the test set FP-MobileNet achieved 99.56% of accuracy. When the number of samples in the training set was smaller than that in the test set, FP-MobileNet still obtained 95.56% of accuracy. The experimental results show that the accuracies of FP-MobileNet under two different data set partitioning schemes are both higher than those of the pure MobileNet model. In addition, FP-MobileNet weighs only occupy 13.7 MB with high recognition rate. It takes into account both accuracy and delay, and is suitable for promotion to mobile devices that require a lightweight model.